{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qFdPvlXBOdUN"
   },
   "source": [
    "# Evaluating ROUGE-L Text Similarity Metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning objectives\n",
    "\n",
    "* Install TF.Text TensorFlow library.\n",
    "* Compute LCS-based similarity score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xHxb-dlhMIzW"
   },
   "source": [
    "## Overview\n",
    "\n",
    "[TensorFlow Text](https://www.tensorflow.org/text/guide/tf_text_intro) provides a collection of text-metrics-related classes and ops ready to use with TensorFlow 2.0. The library contains implementations of text-similarity metrics such as ROUGE-L, required for automatic evaluation of text generation models.\n",
    "\n",
    "The benefit of using these ops in evaluating your models is that they are compatible with TPU evaluation and work nicely with TF streaming metric APIs.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/text_similarity.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MUXex9ctTuDB"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "K_8D_DtQJ0kC",
    "outputId": "e968ff9f-98b0-433a-e951-8db041ac144b"
   },
   "outputs": [],
   "source": [
    "# Install TF.Text TensorFlow library\n",
    "# TODO 1: Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please ignore any incompatibility warnings and errors.\n",
    "\n",
    "**Restart** the kernel to use updated packages. (On the Notebook menu, select Kernel > Restart Kernel > Restart)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "IqR2PQG4ZaZ0"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_text as text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QKp40qS-DGEZ",
    "tags": []
   },
   "source": [
    "## ROUGE-L\n",
    "\n",
    "The Rouge-L metric is a score from 0 to 1 indicating how similar two sequences are, based on the length of the longest common subsequence (LCS). In particular, Rouge-L is the weighted harmonic mean (or f-measure) combining the LCS precision (the percentage of the hypothesis sequence covered by the LCS) and the LCS recall (the percentage of the reference sequence covered by the LCS).\n",
    "\n",
    "Source: https://www.microsoft.com/en-us/research/publication/rouge-a-package-for-automatic-evaluation-of-summaries/\n",
    "\n",
    "The TF.Text implementation returns the F-measure, Precision, and Recall for each (hypothesis, reference) pair.\n",
    "\n",
    "Consider the following hypothesis/reference pair:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "WUgEkGHRKafG"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-04-29 13:33:23.297026: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64\n",
      "2022-04-29 13:33:23.297080: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-04-29 13:33:23.297117: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (tensorflow-2-6-20220429-181606): /proc/driver/nvidia/version does not exist\n",
      "2022-04-29 13:33:23.297465: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "hypotheses = tf.ragged.constant([['captain', 'of', 'the', 'delta', 'flight'],\n",
    "                                 ['the', '1990', 'transcript']])\n",
    "references = tf.ragged.constant([['delta', 'air', 'lines', 'flight'],\n",
    "                                 ['this', 'concludes', 'the', 'transcript']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qeiXnY-_Khp1"
   },
   "source": [
    "The hypotheses and references are expected to be [tf.RaggedTensors](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor) of tokens. Tokens are required instead of raw sentences because no single tokenization strategy fits all tasks.\n",
    "\n",
    "Now we can call [text.metrics.rouge_l](https://www.tensorflow.org/text/api_docs/python/text/metrics/rouge_l) and get our result back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LS_NigzqKgtT",
    "outputId": "7255ed7f-41dc-48ea-b2cf-b677070819b6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F-Measure: tf.Tensor([0.44444448 0.57142854], shape=(2,), dtype=float32)\n",
      "P-Measure: tf.Tensor([0.4       0.6666667], shape=(2,), dtype=float32)\n",
      "R-Measure: tf.Tensor([0.5 0.5], shape=(2,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "result = text.metrics.rouge_l(hypotheses, references)\n",
    "print('F-Measure: %s' % result.f_measure)\n",
    "print('P-Measure: %s' % result.p_measure)\n",
    "print('R-Measure: %s' % result.r_measure)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FQoprhImKoD0"
   },
   "source": [
    "ROUGE-L has an additional hyperparameter, alpha, which determines the weight of the harmonic mean used for computing the F-Measure. Values closer to 0 treat Recall as more important and values closer to 1 treat Precision as more important. alpha defaults to .5, which corresponds to equal weight for Precision and Recall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Q2ZnjOIgKnnS",
    "outputId": "2353f659-bf1e-4425-b6da-0063e3c9be61"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F-Measure (alpha=0): tf.Tensor([0.5 0.5], shape=(2,), dtype=float32)\n",
      "P-Measure (alpha=0): tf.Tensor([0.4       0.6666667], shape=(2,), dtype=float32)\n",
      "R-Measure (alpha=0): tf.Tensor([0.5 0.5], shape=(2,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Compute ROUGE-L with alpha=0\n",
    "result = text.metrics.rouge_l(hypotheses, references, alpha=0)\n",
    "print('F-Measure (alpha=0): %s' % result.f_measure)\n",
    "print('P-Measure (alpha=0): %s' % result.p_measure)\n",
    "print('R-Measure (alpha=0): %s' % result.r_measure)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iYUYiLJhKseb",
    "outputId": "98fdcd02-ff49-421b-928c-773cde0d4248"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F-Measure (alpha=1): tf.Tensor([0.4       0.6666667], shape=(2,), dtype=float32)\n",
      "P-Measure (alpha=1): tf.Tensor([0.4       0.6666667], shape=(2,), dtype=float32)\n",
      "R-Measure (alpha=1): tf.Tensor([0.5 0.5], shape=(2,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Compute ROUGE-L with alpha=1\n",
    "# TODO 2: Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "Tce3stUlHN0L"
   ],
   "name": "text_similarity.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m91",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m91"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
